Skip to main content

Pipeline Sync Modes

While creating a pipelilne, you can choose any of the following sync mode options:

  • Full Refresh|Overwrite: Completely replaces the existing destination data with the latest data from the source.

  • Full Refresh|Append: Adds all records from the source to the destination without modifying or deleting existing records. Ideal for growing datasets.

  • Incremental|Append: Adds only new or updated records from the source to the destination since the last synchronization. This is efficient for large datasets where only recent changes are relevant. A cursor field must be selected for the incremental operations. Acceptable data types are limited to the following data types:

    • Integers

    • longs/numbers

    • Timestamps

  • Full refresh|Merge: This mode performs an upsert (insert or update), adding new records and updating existing ones based on a unique key. An Iceberg format destination is required. You must choose a unique key column. Only applicable for Apache Iceberg format.

  • Incremental|Merge: This mode efficiently updates a target table by processing only changes since the last sync. The specified cursor field or the last updated time is used to perform an upsert (update or insert), appending new records and merging updates to existing ones. Ensure to define a unique key for the table; otherwise, the sync mode defaults to Full Refresh | Incremental Append.
    Acceptable data types are limited to the following data types:

    • Integers

    • longs/numbers

    • Timestamps

warning

  • Incremental|Merge and Full Refresh|Append sync modes are irreversible once saved.

  • For merge operations, you must specify unique columns (primary key) for row identification. If not specified, Incremental|Merge defaults to Incremental|Append

  • Full refresh|Merge and Incremental|Merge are only valid for Iceberg format. If Parquet is selected with merge operations, a warning is displayed.

  • Pipelines can transition from Full Refresh to any other mode. However, once it is in Incremental|Append, Full Refresh|Merge, or Incremental|merge modes, it cannot be reverted to full refresh.

Choosing a unique key column for the table for merge operation​

  1. On the Streams list, from the Output table name column, select the table link.

  2. In the table details page, click the Column info tab.

    Open image-20250224-135624.png

    sample

  3. Click and select Mark as Unique Column option.

    sample